DOCUMENT RESUME 



ED 436 580 



TM 030 442 



AUTHOR 

TITLE 

PUB DATE 
NOTE 



PUB TYPE 
EDRS PRICE 
DESCRIPTORS 



IDENTIFIERS 



Kacer, Barbara A.; Craig, James R. 

Evaluation of Educational Technology in Kentucky Middle 
Schools . 

1999-11-00 

2 5p . ; Paper presented at the Annual Meeting of the Mid-South 
Educational Research Association (Point Clear, AL, November 
17-19, 1999) . 

Reports - Evaluative (142) -- Speeches/Meeting Papers (150) 

MF01/PC01 Plus Postage. 

♦Academic Achievement; Educational Change; Educational 
Research; *Educational Technology; *Maps,\ *Middle Schools; 
Program Evaluation; *Technological Advancement 
♦Kentucky; Kentucky Education Reform Act 1990; Reform 
Efforts 



ABSTRACT 



The Kentucky Institute for Educational Research has created 
six different Innovation Configuration Component (ICC) maps that describe the 
implementation of educational reforms in Kentucky in the context of the 
Kentucky Educational Reform Act. As part of a general study of progress 
toward educational goals in Kentucky, an investigation was conducted of the 
use of the ICC map to describe educational technology used in middle schools 
in Kentucky and, if the map could be used, the nature of the relationship 
between the level of implementation of educational technology and Kentucky's 
high- stakes assessment of academic achievement. Ten middle schools 
representing 5 levels of performance were followed longitudinally, and data 
from the ICC technology map compared with the results were of teacher 
interviews. In the sample of schools studied, there was a positive 
relationship between the degree of implementation of education technology in 
a school and achievement gains posted by the school's students, but it could 
not be determined whether the gains were a function of the implementation of 
the technology. The implementation of educational technology may be a marker 
of a school's willingness to implement reforms. Findings also suggest that 
another iteration of the ICC Map of Educational Technology would be 
appropriate to reflect the actual situation in the schools more accurately. 
(Contains 2 figures, 6 tables, and 12 references.) (SLD) 
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Introduction 



The Kentucky Educational Reform Act (KERA) of 1 990 is a massive education reform 
bill that specified many new initiatives to be developed and implemented to improve the system 
of public schooling in the state of Kentucky. The effectiveness of those initiatives has been 
evaluated in a variety of ways by different public and private organizations and individuals. One 
approach is to study the long term implementation/effects of the initiatives. The Kentucky 
Institute for Education Research, a private, non-profit research organization, has supported a 
three-year longitudinal study to investigate the education innovations being implemented in a 
small sample of middle and high schools across the state of Kentucky. 

The Concerns Based Adoption Model (CBAM), a model of educational innovation and 
change described by Hall and Hord (1987), has been used in this and previous studies (Craig, 
1997; Craig & Kacer, 1999; Craig, Kacer, & Evans, C.S., 1998; Craig & Pankratz, 1996) to 
provide the theoretical framework for the examination of the implementation of reform 
initiatives. One of the diagnostic tools used in CBAM to inform and describe the change process 
is termed an Innovation Configuration Component (ICC) Map. These maps are designed to 
specify the key components associated with an innovation and to define what “use” means in the 
context of implementing an innovation. Through the Kentucky Institute for Education Research, 
six different ICC Maps have been created (KIER, 1996a, 1996b, 1996c, 1996d, 1996e, 1996f, 
1996g) to describe the implementation of education reform initiatives in Kentucky regarding: 

■ professional development of the school staff 

■ extended school services 

■ school-based decision making councils 

■ high school restructuring 

■ family resource and youth service centers 

■ educational technology 

■ the primary program 

For example, on the ICC Map for Professional Development (KIER, 1995), several key 
innovation components have been identified (e.g., “Teachers have an individual professional 
growth plan.”) The descriptive categories associated with this component are: 



(a) 


(b) 


(c) 


(d) 


(e) 


All staff have 


All staff have 


Staff have 


School staff are 


the school has not 


developed 


developed 


developed 


encouraged to 


addressed the 


individual 


individual growth 


individual growth 


develop and submit 


process of 


professional growth 


plans, and a process 


plans, but the 


individual plans to 


integrating 


plans that integrate 


for integrating 


school has not 


the principals or 


individual 


school goals and 
their professional 
career goals. Plans 
are part of a formal 
school process. 


individual plans 
with school goals is 
in beginning stages. 


integrated the 
process of 
individual growth 
plans into 
professional 
development. 


committee. 


professional growth 
plans. 
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Of the five categories associated with the individual growth plan component, category (a) is 
considered the ideal implementation description. In contrast, category (e) describes the least 
desirable implementation of the component with the other categories associated with other points 
on the continuum. Therefore, if through interviews and observations it is determined that 
category (a) best describes how the reform initiative of professional development operates in a 
school, then that school is considered to have completely implemented that particular component 
of professional development. 

The maps serve as the basis for an interview (or interviews) with knowledgeable 
administrators and/or teachers and, in some instances, direct observation to gain information 
about the implementation of key features associated with particular KERA initiatives. Based on 
the interview(s) and direct observations (if any), the interviewer determines the degree of 
implementation of components associated with an innovation. An innovation profile or score 
total can then be determined for each school regarding each initiative. 

Purpose 



The general intent of the three-year longitudinal study has been to: 

■ Identify key factors that differentiate schools making progress toward 
achievement goals from those which are not 

■ Identify key factors that contribute to successful intervention in low performing 
schools and/or schools in crisis 

The particular focus of the research effort reported here was two-fold. First, the intent 
was to assess whether the ICC Map for Educational Technology (KIER, 1995) developed by 
KIER could be effectively used to describe educational technology employed in instruction in 
middle schools in Kentucky and, second, if the map could be used, the nature of the relationship 
between the level of implementation of educational technology and the state’s high-stakes 
assessment of a school’s students’ academic performances as reflected in the state derived 
Accountability Index (see Instrumentation below). 

Sample 

The sample that was being followed longitudinally consisted of ten middle schools from 
across the state of Kentucky. The schools sampled represent five levels of school performance 
based on the initial pattern of change from 1993 to 1995 of each school’s Kentucky Instructional 
Results Information System (KIRIS) Accountability Index (see below) for students’ academic 
performances. There levels were: 



Level 1. (Schools Moving Up) Schools with improving KIRIS scores which had 
baseline KIRIS scores in the upper quartile of all middle or high schools assessed. 
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■ Level 2. (Schools Not Moving Up) Schools with no improvement in KIRIS 
scores which had baseline KIRIS scores in the upper quartile of all middle or high 
schools assessed. 

■ Level 3. (Schools Moving Up) Schools with improving KIRIS scores which had 
baseline KIRIS scores approximately equal to the mean of all middle or high 
schools assessed. 

■ Level 4. (Schools Moving Up) Schools with improving KIRIS scores which had 
baseline KIRIS scores in the lower quartile of all middle or high schools assessed. 

■ Level 5. (Schools Not Moving Up) Schools with no improvement in KIRIS 
scores which had baseline KIRIS scores in the lower quartile of all middle or high 
schools assessed. 

The KIRIS accountability indices for the baseline and each year since 1 993 for the ten 
middle schools in the sample as determined by the state are presented in Table 1 ; the average 
KIRIS Accountability Indices for each Level are presented in Figure 1 . Because of a change in 
the state accountability system, a KIRIS Accountability Index will not be determined for 1998. 
Furthermore, the extent to which the schools as these various levels are representative of all 
middle schools in the state with similar KIRIS Accountability Index values is not known. School 
#3 declined to participate in the third round of data collection. 



Table 1. The KIRIS Accountability Index values and initial 
performance levels for the ten middle schools sampled. 





School 


Level 


Base 


Index 

93 


Index 

94 


Index 

95 


Index 

96 


Index 

97 


Index 

98* 


1 


Ll 


43.5 


38.8 


48.1 


55.1 


50.9 


54.8 




2 


LI 


45.0 


42.4 


47.2 


51.5 


46.7 


47.6 




i 


L2 


453 


45,6 


4L8 


41.6 


37.7 


50.5 




4 


L3 


40.9 


33.4 


48.2 


56.3 


55.7 


55.8 




5 


L3 


38.6 


33.6 


44.1 


51.0 


45.0 


46.6 




6 


L4 


33.9 


31.3 


36.7 


44.3 


36.6 


39.9 




7 


L4 


32.8 


29.4 


36.1 


36.2 


30.6 


36.1 




8 


L4 


34.0 


29.6 


38.5 


45.1 


39.8 


36.6 




9 


L5 


33.3 


33.5 


33.1 


32.7 


30.0 


31.2 




10 


L5 


31.0 


31.9 


29.9 


34.4 


35.5 


32.4 





*The KIRIS Accountability Indices were not calculated for 1998 by the Kentucky Department of 

Education due to changes in the state’s accountability system. 

Scho ol #3 dec lined to* partic ipa te in th e th ird rou nd of data .collection, 



Figure 1 . Accountability Indices for Middle Schools 
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Instrumentation 



The ICC Map for Educational Technology 

First, an attempt was made to use the ICC Map for Educational Technology (KIER, 
1996f) to assess the instructional use of technology in the middle schools. A copy of the map is 
not attached but an example of an item on the map is presented in Table 2. [The map is available 
from the Kentucky Institute for Education Research at the University of Kentucky in Lexington, 
Kentucky.] In general, the map proved to be too unwieldy to be employed effectively since it 
required much detailed information from a school’s educational technology coordinator and/or 
principal regarding whole school use of technology. An example would be the item dealing with 
the integration of technology into the curriculum (refer to Table 2). 

For reasons of expediency during the third year of the longitudinal study the researchers 
reduced the educational technology map to two questions from the original map (refer to Table 
3). In the context of the longitudinal study, the data gained from those two map items were 
“informed” by teacher interview data gathered from teachers in the school regarding their use of 
instructional educational technology. 

(KIRIS) Accountability Index 

The state of Kentucky has developed the Kentucky Instructional Results Information 
System (KIRIS) to determine a school’s Accountability Index (see below) for students’ academic 
performances. [The system is currently undergoing revision and a new system for determining 
an accountability index for a school will be created.] In the past, a school’s Accountability 
Index has been created by the state by combining the assessment of the academic achievement of 
students attending a school with several “non-cognitive” factors (e.g., drop-out rate) into one 
measure of a school’s overall performance. The KIRIS Accountability Index values for schools 
are made generally available to the public by the Kentucky Department of Education. 
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Data Collection Procedures 



During the third year of the longitudinal study, the education technology coordinators in 
nine of the middle schools in the sample were interviewed on site using the modified ICC Map 
for Educational Technology. In most instances, facilities were observed as part of the interview. 
One middle school (a Level 2 school) declined to participate in this round of data collection. 

Findings 



Education Technology 



The modified ICC Map for Educational Technology is scored from 4 for Category (a) to 
1 for Category (d) for Item #1 and from 5 for Category (a) to 1 for Category (e) for Item #2. 
Therefore, the education technology implementation scores vary from a maximum of 9 (4 for 
Item #1 and 5 for Item #2) to a minimum of 2 (1 for Item #1 and 1 for Item #2). The data for the 
middle schools sampled are presented in Table 4. It is noteworthy that the highest scoring school 
(i.e., School #3 which received a 9 on the map) is not a high socioeconomic school. However, 
the district has dedicated the resources necessary to wire the school for a computer network and 
Internet access, provide enough computers to meet the state guidelines ratio of one computer for 
every six students, provide a good library of video materials (tapes and disks) and sufficient 
video equipment, provide necessary scientific equipment for the science curriculum, and so on. 

In addition, almost all of the teachers in the school regularly include the use of word processing, 
use of the Internet, and spreadsheets in teaching all content areas across the curriculum. In 
contrast, in School #8, which scored a 3 on the modified ICC Map for Educational Technology, 
teachers are generally bound to using print materials in their instructional activities. Generally, 
they feel this is necessary in order to keep it simple so students can learn the basic skills they 
need. The teachers at School #8 do not seem to be interested in incorporating much technology, 
especially computer technology, into the curriculum. While there is a computer in each 
classroom for teacher use, they are rarely, if ever, used for student instruction. The library in 
School #8 has a reasonable collection of print, video, and audio materials and equipment and 
provides access to the Internet for student research. The building has not been wired and 
physically adapted to integrate computer or other technologies into the building’s physical plant. 
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Table 4. School educational technology map 
implementation score and associated comments for the middle schools sampled. 



School 

# 


Education 

Technology 

Implementation 

Score* 


Comments 


1 


7 


Has professional development for teachers on 
educational technology during teacher planning 
periods during the school day. 


2 


4 


Has reasonable technology resources in the school for 
teachers; very uneven use of technology among the 
teachers across the school. 


3 


8 


Several classes are technology based. 


4 


9 


Has a technology rich environment; teachers worked 
collectively to incorporate technology into the 
school’s curriculum. 


5 


3 


Teachers have to be trained in the use of technology 
and to be encouraged to incorporate technology into 
the curriculum. 


6 


8 


Makes an effort to use technology in the extended 
school services program. 


7 


4 


Has a great computer laboratory set up & the principal has a 
plan; working to incorporate technology into the 
curriculum & are about 2-3 years away. 


8 


3 


Teachers are focused on “basics” & are not into 
technology; school needs to be physically 
technologically updated. 


9 


3 


Several teachers are positively motivated to 
incorporate technology into their classrooms but the 
technology available is old & is a general state of 
disrepair. 



’"The higher the Education Technology Implementation Score, the closer the implementation of the 
innovation is to what is considered ideal. 
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Student Achievement 



The achievement of the students in a school is reflected in a school’s KIRIS 
Accountability Index as determined by the state. The gain or loss of the ten middle schools’ 
accountability indices is presented in Table 5. Some schools posted significant gains over the 
period (e.g., School #4 gained 22.4 points in its Accountability Index) while others showed 
losses (e.g., School #9 lost 2.3 points) or very minimal gains (e.g., School #10 gained .5 points). 



Table 5. Gain or loss in a school’s KIRIS 
Accountability Index from 1993 to 1997. 



School 


Level 


Index 93 


Index 97 


Index 97- Index 93 
Gain (Loss) 


1 


Ll 


38.8 


54.8 


16.0 


2 


LI 


42.4 


47.6 


5.2 


3 


L2 


45.6 


50.5 4.9 


4 


L3 


33.4 


55.8 


22.4 


5 


L3 


33.6 


46.6 


13.0 


6 


L4 


31.3 


39.9 


8.6 


7 


L4 


29.4 


36.1 


6.7 


8 


L4 


29.6 


36.6 


6.0 


9 


L5 


33.5 


31.2 


(-2.3) 


10 


L5 


31.9 


32.4 


.5 



Relationship between Accountability Index GaintLossl 
and Educational Technology Implementation Score 

The relationship between a school’s Accountability Index gain (or loss) over the five year 
period from 1993 to 1997 and a school’s education technology implementation score as 
determined by the modified ICC Map for Education Technology was examined via a rank-order 
correlation. Even though the sample size was small, the correlation was found to be positive and 
significant (rho=693, p<.05). The relationship is visually depicted in Figure 2. 
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Discussion 



At least in the small sample of middle schools studied, there is positive relationship 
between the degree of implementation of education technology in a school and the achievement 
gains posted by the school’s students. However, it is not known if the gains are directly a 
function of the implementation of education technology in a school or simply associated with the 
attitudes and teaching behaviors of teachers in middle schools which have a high degree of 
implementation of education technology. It is possible that the degree of implementation of 
education technology in a school is a barometer of a school’s general willingness to implement 
the a particular KERA reform initiative and not a function of education technology per se. 

The findings also suggest that another iteration in the formulation of the ICC Map of 
Educational Technology is appropriate. Additional information regarding a school’s 
implementation of education technology would be helpful in better understand the relationship, if 
any, between degree of implementation of education technology in a school and the achievement 
gains posted by the school’s students. Therefore, the researchers have produced a third ICC map 
to assess the use of educational technology in instruction in middle schools. A copy of this map 
is contained in Table 6. Its key features include components that address the infusion of 
educational technology into the curriculum and its impact on assessment. 
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Computer technology instruction is accomplished in a laboratory environment where each classroom teacher participates in the instruction along with 
the students (i.e., it is not a "drop off 5 arrangement). 
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The extent to which teachers/the school has/have modified the curriculum in response to advances in technology. (CURRICULAR CHANGE) 
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